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ABSTRACT 

This paper presents a model for continuous, 
systematic evaluation of ongoing programs and specifies data needed 
for decisionmaking in the areas of program adoption, curtailment, or 
expansion. These data give the decisionmaker the necessary 
information for program planning in today’s sophisticated educational 
environment. (Author/RA) 
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Analysis has been misinterpreted by educators— and even maligned. I think 
there are three reasons for this. First, we have been schooled in the princi- 
ples of classic research design, and many educators remain hung up in this pro- 
cess, despite its often being inappropriate as an evaluative technique. Second 
analysis has often been represented as being esoteric and requiring the use of 
sophisticated techniques, tremendous information systems and large computers. 
And thirdly, a strong belief that "education" is a human process and cannot be 
quantified. The first two objections are rather obviously more amenable to 
correction. The principles of research design are flexible enough to encompass 
good evaluation design and practice. The more complex techniques of analysis 
and computers are helpful, of course, but so are the simpler techniques. The 
problem really lies in the third point: education cannot be quantified and 

for that reason analysis is inappropriate. This paper addresses this problem 
and suggests ways educators can meet the analysts more than halfway. 

I will agree that some aspects of educational outcome are more difficult 
to quantify than others, but I wi 1 1 suggest that an informed estimate of an 
outcome measure is better than nothing. Before we say that analysis is com- 
plex and unsuitable, let's see what is required for good analysis, what we can 
expect it to do for us, and how a cooperative effort between educator and 
analyst can lead to a better understanding of an educational system as it 
currently operates, and to a systematic exploration of the alternatives avail- 
able to improve the system. The purpose of analysis is to aid in making 
decisions that lead, among other things, to program improvement. Since most 



*Thi s paper was prepared for presentation at the annual meeting of the . 
American Educational Research Association, April 3"7, 1972, in Chicago, Illinois. 

+ Any views expressed in this paper are those of the author. They should not 
be interpreted as reflecting the views of The Rand Corporation or the official 
opinion or policy of any of its governmental or private research sponsors. 

Papers are reproduced by The Rand Corporation as a courtesy to members of its 

staff. 






2 



- 2 - 



decisjons about the implementation of programs to meet educational objectives 
are made at the school district level, we wi 1 1 concentrate on analysis vor 
improvement of programs in a district. 

REQUIREMENTS FOR A GOOD ANALYSIS 

The educator needs to decide what specific question he wants to address, 
needs to state it in a form that can be answered, and needs to know what infor- 
mation is necessary for arriving at that answer. 

Let's use one example that we can develop to illustrate these requirements 
as well as to illustrate the kinds of results we can expect and how the educa- 
tor and the analyst can work together to achieve program improvement. 

Before formulating a question, we need to briefly describe the problems 
facing the district decisionmakers. I think it not unreasonable to postulate 
the following situation: 

o Twenty percent of the students in the district score 
two years or more below national norms in arithmetic. 

o Twenty-three percent of the students in the district 
score two years or more below national norms in reading. 

o Absenteeism and truancy are at an all-time high. 

o A bond issue has just failed to pass. 

o There is talk of a teacher strike because failure of the 
bund issue means no increase in pay, and no money to hire more 
teachers to relieve overcrowded classrooms. 

It goes without saying that all these problems cannot be u.-kled simul- 
taneously even though they are interrelated in a very real sense. It is also 
obvious that solutions cannot be found for all of them simul tanepusly ,. but 
that it is well within the state of the art to tackle some of the basic educa- 
tional problems, subject to external operating constraints-! .e . , the large num 
ber of low achievers. We've already met the first requirement for a good 
analysis: we have determined that there is a specific question that we can 

address . 
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The question might well be stated as "What alternative educational ser- 
vices can be provided by the district so that students in the math and/or 
reading programs achieve better?" We can use either math or reading as an 
illustration because analysis of either program will have the same set of 
reouiraments. Let's also recognize that we are talking about only one of the 
possible objectives of the program--achievement— but any other objective 
could be treated by the same methodology. 

The next requirement for a good analysis is to specify the data that 
will be required to answer the question. In so doing, what we are really 
aiming for is a description of the program as it now exists. This, in turn, 
is a first step toward developing the capability for carrying out an on-going 
systematic evaluation of a districts' educational programs. It is only by 
having a detailed knowledge of what is being done today, with what outcome, 
that we can attack the problem of what needs to be done to improve the pro- 
gram. 

In order to build the base case or to describe the current program in 
arithmetic, for example, what data do we need? Since our concern as educa- 
tors is not with district averages, or with school averages, but with indi- 
vidual students, ideal ly we woul d do our analysis at that level. In the 
interests of reality, however, that will remain for the moment as a goal to 
strive for, except in a very small school district, and we'll consider an 

analysis at the classroom level. 

By themselves, achievement data yield information only about outcome. 
What is needed is a description of the school environment that produced the 
achievement. To answer our question, information is needed about what pro- 
grams are provided, what resources they require, and what achievement they 
produced. Even in a large district, this kind of data can be handled without 
a computer, if it is approached by program. One can envision a simple table 
that displays the basic information. It would essentially show achievement 
data in arithmetic for each classroom, for each grade in each school. It 
would, in addition, show the mean district achievement score for each grade. 

(see Fig. l) 
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What Tan Be Expected From an Analysis 

Before we explore what this kind of simple analysis will do for us, let's 
return for a minute to the stated objections to analysis and re-examine them. 

We said first that educators cling to research design. What we have done here 
is move one step away from classical research design toward the goal of setting 
up an on-going evaluation. We have arrayed the outcome data for a program so 
that we may start to examine the environments that contributed to those out- 
comes. As in good research design, we are looking for differences, but in 
contrast to having controlled on a set of variables before treatment was 
instituted, nothing was artifically arranged, and we are now able to assess 
what happened under varying conditions. 

The second postulated objection to analysis was that it required sophis- 
ticated techniques, large information systems and powerful computers. The 
analysis we have just set up can be done without any such trappings. 

Thirdly, we said that the major obstacle was that education cannot be 
quantified. If we accept the fact that education is a complex system made up 
of many components, and if we are willing to focus on those components, one 
at a time, we should eventually be able to better understand the interrelation- 
ships that comprise the system. For the time being, we should take first 
steps, and try to use analysis to help us solve the most pressing problems, 
one at a time. As educators and analysts work together on individual problems, 
they will both develop the insights necessary to tackle the more complex problems 

Let's come back now to the information we have collected to help us address 
the problem of providing educational services to attain better achievement in 
arithmetic. By looking at an array of scores by classroom, we can quickly get 
a picture of where our attention needs to be concentrated. 

If I were a school district administrator, I would want to focus my atten- 
tion on two kinds of classrooms: those where achievement was better than normal 

and those where it was less than satisfactory. The array of scores suggested 
here allows us to do just that. We are now in a position to ask a series of 
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questions relating to the environment in which the deviant achievement scores 
occurred. We really want to do this for two reasons. If some curriculum or 
teaching mode is succeeding better than others, one of our alternatives for 
improving the program is to encourage wi de-spread adoption of the successful 
technique. Conversely, if a detailed description of each classroom that we 
have isolated for study shows that some technique is not yielding good out- 
comes, there is now good objective data to support some change in the class - 
room. 

f 

It is not my intention to gloss over the large amount of work involved 
in creating a description of each classroom that is meaningful for the task 
at hand. It is my intention to say that it is an effort that can lead to 
improvement in the educational system. Our one quantitative measure is our 
outcome measure--achievement. But, as: was said previously, we need to be able 
to describe the school environment that produced the achievement, because it 
is that environment, or parts of it, that will have to be changed to improve 
the outcome measures. 

A Cooperative Effort Between Educator and Analyst 

l J ve talked about only the simplest kind of analysis. But I think of it 
as a first step toward a systematic way of looking at school programs, and 
considering the alternatives that are available for improving them. We need 
to analyze our problems one at a time, and given limited resources, we should 
probably concentrate on the most pressing problems. As the analyst and the 
educator work together each will bring to the analytic problem insights which 
will eventually give them the capability to consider complex problems and use 
more sophisticated analytic techniques. For the moment, analysts must be 
keenly aware of the fact that educational outcome measures are not particularly 
reliable. Educators must, on the other hand, be ready to accept the simple 
analytic techniques that are appropriate to the level of sophistication of 

their outcome measures. 
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